Predicting the Internet’s Evolution with Decision Trees and Lasso Logistic Regression Models

The Internet self-evolves rapidly and its dynamic structure poses many interesting questions for researchers in network analysis. In this paper I show how we can simplify the entire Internet as a mathematical graph and then extract its structural characteristics; these characteristics in turn help us build statistical models that can predict how the Internet will evolve. The data describing the Internet structure are both clustered and unbalanced. I hence test various models, including lasso logistic regression, gradient-boosted decision trees and random forest decision trees, to see how well they cope with unbalanced and clustered data. The best performing model was created through a gradient-boosted decision tree that balances flexibility in fitting with robustness in prediction. I show that we can achieve good predicting power using fairly simple explanatory variables, but I also discuss how we can extract more sophisticated variables to improve the models’ performance.